Remote direct memory access adapter state migration in a virtual environment

ABSTRACT

In an embodiment of the present invention, a method includes partitioning a plurality of remote direct memory access context objects among a plurality of virtual functions, establishing a remote direct memory access connection between a first of the plurality of virtual functions, and migrating the remote direct memory access connection from the first of the plurality of virtual functions to a second of the plurality of virtual functions without disconnecting from the remote peer.

FIELD

Embodiments of this invention relate to RDMA (remote direct memoryaccess) data transfer in a virtual environment.

BACKGROUND

Traditional RDMA allows data to move directly from one computer systeminto the memory of another without involving either one's CPU (centralprocessing unit), and specifically, either one's operating system,during the data transfer. This permits high-throughput, low-latencynetworking by eliminating the need to copy data between applicationmemory and the data buffers in the operating system.

A virtual computing environment refers to a computer system in which asingle physical machine may be observed as multiple virtual machines,and where a set of physical hardware resources can be used as multiplevirtual resources. Each virtual machine (VM) can run its own operatingsystem that may control the set of virtual hardware resources.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention are illustrated by way of example,and not by way of limitation, in the figures of the accompanyingdrawings and in which like reference numerals refer to similar elementsand in which:

FIG. 1 is a block diagram that illustrates a computing system supportinga virtual environment according to an embodiment.

FIG. 2 illustrates an RDMA connection between a system and a remote peeraccording to an embodiment.

FIG. 3 illustrates a method for establishing and migrating an RDMAconnection according to an embodiment.

FIG. 4 illustrates partitioning RDMA context objects according to anembodiment.

FIG. 5 illustrates an embodiment in which a network adapter includes anonboard memory.

DETAILED DESCRIPTION

Examples described below are for illustrative purposes only, and are inno way intended to limit embodiments of the invention. Thus, whereexamples are described in detail, or where one or more examples areprovided, it should be understood that the examples are not to beconstrued as exhaustive, and embodiments of the invention are not to belimited to the examples described and/or illustrated.

FIG. 1 illustrates a computer system 100 that supports a virtualenvironment. System 100 may comprise virtual machines 110A and 110B,virtual machine monitor 106, hardware resources 110, and logic 130.Logic 130 may comprise hardware, software, or a combination of hardwareand software (e.g., firmware). For example, logic 130 may comprisecircuitry (i.e., one or more circuits), to perform operations describedherein. For example, logic 130 may comprise one or more digitalcircuits, one or more analog circuits, one or more state machines,programmable logic, and/or one or more ASIC's (Application-SpecificIntegrated Circuits). Logic 130 may be hardwired to perform the one ormore operations. Alternatively or additionally, logic 130 may beembodied in firmware, in machine-executable instructions 132 stored in amemory, such as memory 104, to perform these operations, or in variousother components of system 100. Logic 130 may be used to perform variousfunctions by various components as described herein.

System 100 may comprise a plurality of virtual machines 110A and 110B.While only two are shown, system 100 may comprise more or less virtualmachines than those illustrated. Hardware resources 110 may bevirtualized, meaning that a single physical hardware resource 110 may bepartitioned into multiple virtual hardware resources to enable system100 to use the single physical hardware resource 110 in multiple virtualmachines 110A and 110B. Virtualization may be implemented using VMM 106.In an embodiment, VMM 106 comprises software that imposes avirtualization layer in system 100 in which hardware resources 110 maybe virtualized for use by guest software running on virtual machines110A and 110B.

Hardware resources 110 refer to physical hardware components associatedwith system 110 including, but not limited to, a processor 102, such asCPU (central processing unit), host memory 104, network adapter 108, I/O(input/output) device 120, and non-volatile random access memory (NVRAM)122. Processor 102, host memory 104, network adapter 108, I/O device120, and NVRAM 122 may be comprised in one or more integrated circuitsor one or more circuit boards, such as, for example, a systemmotherboard 118. Alternatively, network adapter 108 and/or I/O device120 may be comprised in one or more circuit cards that may be insertedinto circuit card slots.

Processor 102 may comprise processing circuitry to carry outcomputational tasks of system 100. For example, processor 102 mayinclude a CPU such as, for example, the Intel® Pentium® family ofprocessors, or Intel® Core® family of processors, both commerciallyavailable from Intel® Corporation. Of course, alternatively, processor102 may comprise another type of processor, such as, for example, amicroprocessor that is manufactured and/or commercially available fromIntel® Corporation, or a source other than Intel® Corporation, withoutdeparting from embodiments of the invention.

Host memory 104 may store machine-executable instructions 132 that arecapable of being executed, and/or data capable of being accessed,operated upon, and/or manipulated by logic, such as logic 130, and/orprocessor, such as processor 102. The execution of program instructions132 and/or the accessing, operation upon, and/or manipulation of thisdata by logic 130 for example, may result in, for example, system 100and/or logic 130 carrying out some or all of the operations describedherein. Host memory 104 may, for example, comprise read only, massstorage, random access computer-accessible memory, and/or one or moreother types of machine-accessible memories. This may include, forexample, DRAM (dynamic random access memory) or SRAM (static randomaccess memory), but embodiments of the invention are not so limited.

Network adapter 108 as referred to herein relates to a device which maybe coupled to a data transmission medium to transmit data to or receivedata from other devices coupled to the data transmission medium. Forexample, network adapter 108 may be designed to transmit data to orreceive data from devices coupled to a network such as a local areanetwork. Such a network adapter may communicate with other devicesaccording to any one of several data communication formats such as, forexample, communication formats according to versions of IEEE Std. 802.3(Ethernet), IEEE Std. 802.11, IEEE Std. 802.16, Universal Serial Bus,Firewire, asynchronous transfer mode (ATM), synchronous optical network(SONET), synchronous digital hierarchy (SDH), Internet Wide Area RDMAProtocol (iWARP), or InfiniBand standards.

Network adapter 108 may include onboard memory, but may also use hostmemory 104 to store RDMA context objects necessary to maintain state ofmany RDMA connections. FIG. 5 illustrates an embodiment of the presentinvention in which network adapter 108 includes onboard memory 510,which, in one embodiment, may be a 32 KB on-chip random access memory(RAM). Onboard memory 510 may be used for storing segment table 520,where each segment table entry may be used to store an address or otherreference to page table 530 in host memory 104. Each page table entrymay be used to store an address or other reference to one of 4 KB pages540 in host memory 104 where context objects may be stored.Alternatively, one or more segment table entries may be used to store anaddress or other reference to one or more 2 MB pages 550 in host memory104 where context objects may be stored. Other embodiments of thepresent invention may use any variety of memory hierarchies, pages orother memory region sizes, or any other aspects of memory or datastructure organization, and any other arrangement for storing a portionof a data structure in onboard memory 510 and a portion in host memory104.

Furthermore, network adapter 108A may include indexing hardware 108A tomaintain assignments of RDMA context objects, as further describedbelow. Indexing hardware 108A may include control and/or computationcircuitry, storage elements, and/or any other structures to performand/or facilitate the assignment, indexing, referencing, maintenance,and other functions related to RDMA context objects according toembodiments of the present invention.

I/O device 120 may represent any I/O or peripheral device and/or acontroller or adapter for any such device. I/O device 120 may supportI/O virtualization; for example, I/O device 120 may include or provideone or more physical functions (each, a “PF”) that may be controlled bya VMM, each PF providing the capability to provide and one or morevirtual functions (each, a “VF”). In an embodiment in which system 100includes a PCI (Peripheral Component Interconnect) bus based on PCILocal Bus Specification, Revision 3.0, Feb. 3, 2004, I/O virtualizationmay be based on Single Root I/O Virtualization and SharingSpecification, Revision 1.0, Sep. 11, 2007. Other versions of thesespecifications and/or other specifications, as well as other protocols,may be used in embodiments of the invention.

A VMM may configure and manage the physical resources of I/O device 120such that each of the VFs may be controlled and/or accessed directly bya VM. Therefore, a VF supported by I/O device 120 and assigned to avirtual machine may transfer data within, into, or out of system 100,under control of the virtual machine, without the intervention of a VMM.Furthermore, network adapter 108 may support RDMA capability such thatone or more PFs or VFs of I/O device 120 may transfer data out of hostmemory 104 to a system other than system 100 or into host memory 104from a system other than system 100, and/or between virtual machines insystem 100 and virtual machines in systems other than system 100,without involving processor 102.

FIG. 2 illustrates RDMA connection 200 between system 100 and remotepeer system 290 according to an embodiment of the invention. In thisembodiment, VF 202A of I/O device 120 may be assigned to VM 110A.Accordingly, device driver 210A for VF 202A may reside in an area ofhost memory 104 allocated to VM 110A. Device driver 210A may communicatewith network adapter 108 to establish RDMA connection 200 between system100 and remote peer system 290, as further described below. RDMAconnection 200 may be based on an RDMA protocol as set forth, forexample, for example, by Infiniband™ Architecture (IBA), Volume 2,Release 1.2.1., October 2006; or Remote Direct Data Placement (RDDP),RFC 4296, December 2005. Other versions of these specifications and/orother specifications, as well as other protocols, may be used inembodiments of the invention.

FIG. 3 illustrates method 300 for establishing and migrating an RDMAconnection according to an embodiment of the invention, and FIG. 4illustrates partitioning of RDMA context objects according to anembodiment of the invention. The descriptions of FIG. 3 and FIG. 4 maybe facilitated with references to elements of each other and of FIG. 1and FIG. 2; however, embodiments of the present invention are notlimited to those illustrated or described. Blocks of method 300 referredto as performed by software 204 may be performed by any software or anycombination of software running, executing, or otherwise operatingwithin system 100, such as guest software 204A (which, for example, maybe application software and/or operating system software), other guestsoftware, device driver 210A, a device driver for a PF of I/O device120, VMM 106, or any other software or combination of software.

In block 310, onboard memory 510 may be partitioned such that the memoryspace for each RDMA context object supported by network adapter 108 isassigned to a particular VF of I/O device 120 or another particular VFin system 100. For example, as shown in FIG. 5, segment table entries522 may be assigned to VF 202A, such that all RDMA context objectsstored in pages 542 on host memory 104 belong to VF 202A. Theassignments and/or information related to the assignments may be storedin NVRAM 122, which may be reprogrammed such that the assignments may bemodified from time to time.

In block 314, hardware resources related to context objects may bepartitioned among VFs by assigning one or more hardware resources to aparticular VF. In addition or alternatively, one or more hardwareresources may be designated to be shared between or among two or moreVFs. Shared hardware resources may not be referenced by context objects,because all context objects are assigned to a single VF. The assignmentsand/or information related to the assignments may be stored in NVRAM122, which may be reprogrammed such that the assignments may be modifiedfrom time to time.

In block 318, guest software 204A may perform one or more operationsthat result in network adapter 108 setting up RDMA context objects suchas queue pair (QP) 406, send queue 408A, receive queue 410A, andcompletion queue 412A. QPs are set up such that each send queue has acorresponding receive queue on the remote peer. For example, QP 406 isset-up such that send queue 408A corresponds to receive queue 410B onremote peer 202B, and send queue 408B on remote peer 202B corresponds toreceive queue 410A. Within embodiments of the present invention, anynumber of these context objects may be used for any number of RDMAconnections, along with any number of other RDMA context objects.

In block 322, guest software 204A may initiate a data transfer betweenVF 202A and any memory, device, or other remote peer 202B within remotepeer system 290 by generating a data transfer request. For example,guest software 204A may generate send request 414A, and post sendrequest 414A to send queue 408A. Send request 414A may includeinformation about the data transfer, such as a memory, memory-mappedI/O, or other address at which the data is to be fetched and/or stored,a request type (e.g., SEND), and the length of the data transfer. Anynumber of requests may be generated and posted.

In block 326, data transfer from VF 202A to remote peer 202B may begin,for example, by beginning the data transfer specified by the informationin send request 414A.

In block 330, the assignment of all context objects to a particular VFis maintained, for example, by keeping an indexed list of contextobjects and their assignment to a particular VF. For example, inembodiments with shared and unshared resources, context objects relatedto resources that are not shared or partitioned start with an index ofzero (e.g., each VF may have a QP with an index of zero). However, forcontext objects that relate to resources that are shared or partitioned,a combination of function, object type, and object index may be used toaddress host memory, where indexing hardware 108A performs a translationor other computation to maintain the appearance, to software 204, thatzero-based indexing is being used. For example, a particular contextobject may appear to software 204 to have an index of zero, but theactual non-zero index may be determined by indexing hardware 108A basedon a non-zero index number for a VF, where each non-zero index numberfor each VF may be based on the number of entries for one or more VFs.Block 330 is performed such that pages or other areas of host memory 104used to store context objects are not shared between different VFs.

In block 334, a determination is made to migrate RDMA connection 200from VF 202A to VF 202C. The migration may be desired based onapplication needs, load balancing, or any other reason.

VF 202C may be a VF of I/O device 120 or of any other I/O device withinsystem 100 or another system, and/or may be assigned to VM 110A or anyother VM within system 100 or another system. Although VF 202C is shownas within system 100, VF 202C may be in another system.

Furthermore, VF 202C may use network adapter 108 (e.g., VF 202A and VF202C use different ports on the same adapter) or any other networkadapter within system 100 or any other system, such that the migrationof RDMA connection 200 may or may not include a migration from networkadapter 108 to any other network adapter within system 100 or anothersystem that operates according to an embodiment of the presentinvention.

In block 338, a check is performed on VF 202C and its network adapter todetermine if sufficient resources are available for the migration.

In block 342, the flow of inbound and outbound packets over RDMAconnection 200 is stopped. Inbound packets may be dropped, but thedropping of these packets may be detected and handled according to anyknown techniques. Outbound packets may be flushed to the network.

In block 346, any state related to VF 202A and/or VM 110A may be flushedfrom network adapter 108 to host memory 104.

In block 350, memory space may be allocated for context objects for VF202C. In an embodiment wherein VF 202C is in system 100, the memoryspace may be allocated within host memory 104. In an other embodiment,the memory space may be allocated within the host memory of anothersystem.

In block 354, the contents of host memory 104 that is used for contextobjects previously assigned to VF 202A is copied, using CPU copy, DMA,RDMA, or any known technique, to the host memory space allocated forcontext objects for VF 202C. Therefore, the re-assignment of contextobjects from VF 202A to VF 202C may be transparent to guest software204A or any other operating system, application software, or othersoftware executing on behalf of a user of system 100 or remote peersystem 290.

In block 360, parameters necessary or useful for RDMA connection 200,such as a local network address and network routing configurationinformation, may be transferred to device driver 210C for VF 202C.

In block 364, the host memory space allocated to VF 202C is associatedwith VF 202C.

In block 368, the flow of packets over RDMA connection 200, now betweenVF 202C and remote peer 202B, is continued.

Method 300 may include any additional operations desired, such as thoseinvolving known RDMA techniques. Furthermore, blocks illustrated inmethod 300 may be omitted and/or reordered within the scope of thepresent invention.

In the foregoing specification, the invention has been described withreference to specific embodiments thereof. It will, however, be evidentthat various modifications and changes may be made to these embodimentswithout departing therefrom. The specification and drawings are,accordingly, to be regarded in an illustrative rather than a restrictivesense.

What is claimed is:
 1. A method comprising: partitioning a plurality ofremote direct memory access context objects among a plurality of virtualfunctions; establishing a remote direct memory access connection betweena first of the plurality of virtual functions and a remote peer using anetwork adapter; and migrating the remote direct memory accessconnection from the first of the plurality of virtual functions to asecond of the plurality of virtual functions without disconnecting fromthe remote peer.
 2. The method of claim 1, further comprisingpartitioning a plurality of hardware resources related to the pluralityof remote direct memory access context objects among the plurality ofvirtual functions.
 3. The method of claim 1, wherein partitioning theplurality of remote direct memory access context objects includespartitioning a memory space into a plurality of regions, each region tobe assigned to one of the plurality of virtual functions for storage ofa portion of the plurality of remote direct memory access contextobjects.
 4. The method of claim 3, further comprising storing a segmenttable in a segment table space in the network adapter.
 5. The method ofclaim 4, further comprising storing the plurality of remote directmemory access context objects in a host first memory, each at a locationreferenced by an entry in the segment table.
 6. The method of claim 5,further comprising assigning each entry in the segment table to one ofthe plurality of virtual functions.
 7. The method of claim 6, furthercomprising allocating a first space in the first host memory for thefirst of the plurality of virtual functions.
 8. The method of claim 7,further comprising storing a first of the plurality of remote directmemory access context objects in the first space.
 9. The method of claim8, further comprising allocating a second space, in one of the firsthost memory and a second host memory, for the second of the plurality ofvirtual functions.
 10. The method of claim 9, wherein migrating includescopying contents of the first space to the second space.
 11. A systemcomprising: an input/output device including a first of a plurality ofvirtual functions; and a network adapter to establish a remote directmemory access connection between the first virtual function and a remotepeer, and to migrate the remote direct memory access connection from thefirst virtual function to a second virtual function withoutdisconnecting from the first peer.
 12. The system of claim 11, furthercomprising a host memory to store a plurality of remote direct memoryaccess context objects to be partitioned among the plurality of virtualfunctions.
 13. The system of claim 11, further comprising a plurality ofhardware resources related to the plurality of remote direct memoryaccess context objects to be partitioned among the plurality of virtualfunctions.
 14. The system of claim 11, wherein the network adapterincludes a segment table memory space to be partitioned among theplurality of virtual functions.
 15. The system of claim 14, wherein thesegment table is to store a plurality of segment table entries, eachsegment table entry to refer to one of a plurality of locations in thehost memory, each location to store at least one of the remote directmemory access context objects.
 16. The system of claim 15, wherein thenetwork adapter is to copy contents of a first of the plurality oflocations to a second of the plurality of locations, the first of theplurality of locations assigned to the first virtual function and thesecond of the plurality of locations assigned to the second virtualfunction.
 17. An apparatus comprising: a network adapter to establish aremote direct memory access connection between a first of a plurality ofvirtual functions and a remote peer, and to migrate the remote directmemory access connection from the first virtual function to a secondvirtual function without disconnecting from the first peer; and anonboard memory to store a segment table to be partitioned among theplurality of virtual functions.
 18. The apparatus of claim 17, whereinthe segment table is to store a plurality of segment table entries, eachsegment table entry to refer to one of a plurality of locations in ahost memory, each location to store at least one of a plurality ofremote direct memory access context objects.
 19. The apparatus of claim18, wherein the network adapter is also to copy contents of a first ofthe plurality of locations to a second of the plurality of locations,the first of the plurality of locations assigned to the first virtualfunction and the second of the plurality of locations assigned to thesecond virtual function.
 20. The apparatus of claim 18, furthercomprising a plurality of hardware resources related to the plurality ofremote direct memory access context objects to be partitioned among theplurality of virtual functions.